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■  Introduction 

■  Improving  Existing  Techniques 
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TD-1-12  Description 


■  Goal  is  to  reduce  schedule  slip  and  cost  growth  due 
to  immature  technology  by 

■  Reducing  the  likelihood  that  immature  technology  is 
accepted  into  acquisition  programs 

Or 

■  Better  revealing  upfront  the  risks  associated  with 
accepting  immature  technology 
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TD-1-12  Approach,  Products 


Two-pronged  Approach 

■  First,  improve  existing  techniques  for  gauging  maturity 

■  Improve  quality  &  consistency  of  AF  Technology  Readiness  Level 
(TRL)  assessments  via  improved  training 

■  e.g.  Help  reduce  the  questions  on  what  a  “relevant  environment”  is 

■  Applicable  to  formal  TRA  process  as  well  as  organic  program  office  TRL 
assessments 

■  Produce/promulgate  training  for  Manufacturing  Readiness 
Assessments  (MRA) 

■  Improve  software  tech  readiness  level  descriptions  ->  OSD 

■  Second,  produce  methodology  to  evaluate  integration  and  the  ‘ilities 

■  “Risk  Identification:  Integration  &  ilities”  (RI3)  methodology 
identifies  sources  and  categories  of  technical  risks  in  developing 
and  incorporating  new  technologies 

■  RI3  Guidebook 

■  RI3  questionnaire  tool 
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Better  Assessment  of  Technology 

Readiness  Levels 


■  Assessments  of  Technology  Readiness  Levels  (TRLs)  occur  for  large 
and  small  programs 

■  Small  programs  ->  Organic  SPO  resources 

■  Large  programs  ->  Receive  OSD  oversight,  involve  independent 
assessment  teams  (TRA  process) 

■  Improved  training  should  lead  to 

■  More  consistent  results  in  Air  Force  programs,  independent  of 
program  size 

■  Less  time  spent  discussing  definition  of  “relevant  environment” 
for  a  particular  technology 

■  Cases  studies  being  assembled  into  a  new  training  class 

■  Audience:  subject  matter  experts  who  will  need  to  conduct  TRL 
assessment 

■  Based  on  recent  SAF/AQR  TRA  improvements  plus  assessments 
of  programs  visited  by  this  study 

■  Looking  for  course  developer  to  assist  AFIT 


Integrity  -  Service  -  Excellence 


6 


Software  (Technology)  Readiness 

Levels 


■  Background 

■  Current  definitions  of  Software  Technology  Readiness 
Levels  in  DoD  TRA  handbook  have  lead  to  confusion 

■  Software  subteam  of  TD-1-12  identifying  shortfalls  in  current 
definitions  &  formulating  potential  changes 

■  Working  in  conjunction  with  AQR-funded  effort  at  CMU/SEI 
to  identify  issues 

■  Goal 

■  Using  the  current  definitions,  frame  enough  supporting 
materials  to  make  the  definitions  useful 

■  Draft  output  in  August  08 

■  Future  goal  (if  necessary):  fix  the  definitions 
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Training  For  Manufacturing 
Readiness  Assessments 
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■  MRAs  fill  the  vital  role  of  predicting  whether  or  not  we  will  be 
able  to  produce  the  product  in  the  timeframe  and  at  the  rate 
desired  with  the  desired  quality 

■  Identifies  risks  for  a  program  office  to  work  on 

■  Policy  in  development  currently 

■  Air  Force  Institute  of  Technology  is  developing  training  and 
tools  to  enable  MRAs  to  be  conducted  at  each  of  its  four 
product  centers,  based  on  AFRL  Mantech  team’s  work. 


Integrity  -  Service  -  Excellence 


8 


TD-1-12  Outline 


■  Introduction 

■  Improving  Existing  Techniques 

■  Methodology  for  Integration  &  ilities 

■  Worksheet/Tool  Concepts 

■  Summary 
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Measuring  Technology  Readiness 

(DoD  TRA  Deskbook,  May  2005) 


System  Test,  Launch 
&  Operations 


Syste  m/S  u  bsyste  m 
Development 


TRL  9 

TRL  8 

TRL  7 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


Basic  Technology 
Research 


TRL  6 

TRL  5 


TRL  4 

TRL  3 


Technology  Readiness  Levels  ( TRLs ) 

9.  Actual  system  proven  through  successful  mission 
operations  (sw  mission-proven  operational 
capabilities) 

8.  Actual  system  completed  and  qualified  (sw  mission 
qualified)  through  test  and  demonstration  (sw  in 
an  operational  environment) 

7.  System  prototype  demonstration  in  an  operational 
(sw  high-fidelity)  environment 

6.  System/subsystem  model  or  prototype 

demonstration  in  a  relevant  environment  (sw 
module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment) 

5.  Component  and/or  breadboard  (sw  module  and/or 
subsystem)  validation  in  relevant  environment 

4.  Component  and/or  breadboard  validation  in 
laboratory  environment 

3.  Analytical  and  experimental  critical  function  and/or 
characteristic  proof-of-concept 

2.  Technology  concept  and/or  application  formulate 

1.  Basic  principles  observed  and  reported 
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Goals  of  New  Integration  /  ‘ilities 

Criteria 


■  TRL  tells  you  where  you  are,  but  is  not  an  indicator 
of  future  success 

■  Data  shows  that  programs  reaching  MS  B  with  TRL  5 
or  6  fare  no  better  (7  does  fare  better) 

■  Need  a  complementary  methodology  to  give 
program  offices  better  ways 

■  To  avoid  common  pitfalls 

■  To  report  upwards  (eg  in  PoPS) 
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Surveyed  Globe  for  Good  Ideas 


■  Efforts  surveyed  across  DoD,  other  agencies,  internationally, 
universities,  corporate  world 

■  NASA-originated  AD2  methodology  viewed  favorably  by 
members  in  OSD  AT&L  and  SAF  AQR 

■  British  Ministry  of  Defence  provided  good  input 

■  British  System  Readiness  Levels  (SRLs)  are  used  in 
conjunction  with  TRLs 

■  Also  in  conjunction  with  a  full-blown  risk  analysis  assessment 
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Note: 

Each  box  on  the  matrix  represents  a  Key 
Output forthat  system  discipline.  The 
colours  represent: 

Green:  full  achievement  of  the  required  outputs 
Amber:  some  shortfalls  in  the  required  outputs 
Red:  significant  shortfall  in  the  required  outputs. 


Example  of  an  5RL  ‘Signature’ 
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Examined  Case  Studies  and 
Formed  Opinions 


■  Conducting  case  histories  on  5  current  and  historical 
programs  at  product  centers  and  1  at  NASA 

■  Mix  of  air  and  space  projects  (no  cyber-only) 

■  Program  literature  (eg  quarterly  DAES  reports) 

■  Live  interviews 

■  Combined  case  histories  with  team  members’  knowledge  to 
form  lessons  learned  and  identify  best  practices 

■  Plan  to  “test”  and  refine  methodology  by  using  historical  data 
from  another  set  of  programs 

■  Final  judgement:  The  issues  that  are  lacking  with  TRL 
assessments  are  not  where  you  are  but  what  are  the  issues 
lying  ahead 

■  No  new  scales  required  (no  IRL,  SRL,  etc.) 

■  Identification  of  risks  is  the  key  (as  is  done  in  MRAs) 
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How  to  Begin  RI3  Methodology 


■  Start  with  system  level  gross  evaluation  (top-down) 

■  Break  down  into  subsystems,  note  Critical  Technology  Elements 
(CTEs),  and  evaluate  TRL  at  appropriate  level 

■  To  assess  integration  and  ‘ilities,  must  evaluate  CTEs  +  units  that 
interface  with  CTEs,  even  if  they  are  not  CTEs  themselves 

■  Then,  proceed  back  up  tree  as  appropriate 
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‘ilities  Threads 


Team  has  downselected  to  the  following  list* 

■  People,  organization,  &  skills 

■  Design  Maturity  and  Stability  (stability  of  rqmts) 

■  Scalability  &  Complexity 

■  Reliability 

■  Maintainability 

■  Software 

■  Human  factors 

■  Integrability 

■  Testability 

List  culled  from  INCOSE  standards  and  driven  by 
observations  of  past  program  problems 


Some  graphics  on  other 
charts  are  out  of  date 
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Some  Sample  Questions 


■  Integrability 

■  Are  there  interactions  /  integration  issues  that  could  be  affected 
by  proprietary  or  trust  issues  between/  among  suppliers? 

■  Have  key  sub-systems,  at  whatever  level  of  readiness 
(breadboard,  brassboard,  prototype),  been  tested  together  in  an 
integrated  test  environment  and  have  they  met  test  objectives? 

■  Software 

■  Are  personnel  with  development-level  knowledge  of  the  existing, 
reused  software  part  of  the  new  software  development  team? 

■  Maintainability 

■  Is  modeling  and  simulation  used  to  simulate  and  validate 
maintenance  procedures  for  the  unit  under  test  and  higher  levels 
of  integration? 

■  Explanatory  discussion  with  potential  best  practices  on  each 
question  are  included  in  RI3  guidebook  and  Excel-like 
worksheet/tool 

■  Questions  are  technical  and  shy  away  from  programmatic 

■  Approximately  90  questions  under  development  (~10  per  ’ility) 
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Development  of  Risks 


■  Questions  contain  a  best  practice  and  are  meant  to  prompt  a 
program  manager  to  consider  acting  accordingly 

■  Questions  may  be  answered  “Yes,”  “No,”  or  “not 
applicable” 

■  If  the  answer  is  no,  then  the  next  step  is  to  identify  & 
describe  risks  that  may  result 

■  Risks  are  compiled  and  then  rated  using  standard 
likelihood,  consequence  methodology 

■  Methodology  assumes  typical  systems  engineering  processes 
are  in  place  (Systems  Engineering  Assessment  Model  [SEAM] 
applied) 

■  Eliminates  need  for  most  process  questions  from  RI3 
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Assess  Likelihood  and 
Consequence  for  Each  Risk 

■  Utilize  “standard”  DoD/AF  definitions 
for  “Likelihood”  and  “Consequence” 

■  Lg[1,5] 

■  Ce[1,5] 

■  2-Dimensional  plot  has  defined 
R,Y,G  colors 

■  For  each  question,  can  plot  results  of 

the  risks  that  are  spawned  ^ 

■  Each  ‘ility  area  has  a  different  o 

spread  on  its  own  scatter  plot  £ 

■ 

■  Produces  9  scatter  plots  for  a  0 

UUE  - 


■  Utility 

■  Within  a  thread,  concentrates 
program  manager  on  area 
(question)  that  needs  work 

■  L,C  outputs  should  be  used  as 
inputs  to  a  risk  assessment 
process 


Example  Results: 
Integrability  for  UUE 
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Why  Summarize  Each  ‘ility  Area? 


■  Manager  of  the  Unit  Under  Evaluation  (UUE)  is  left 
with  8  or  9  separate  scatter  plots 

Integrability  Testability  Reliability  Etc. 
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J 

a  IT 

lb 

ig 
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2d 

2e 
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2c 

2f 

3c  3d 

3a 

3b  3e 

3f 

::::::::: 

■  Summarization  of  the  details  would  improve 

■  Understanding  of  overall  status 

■  Reporting  upwards 
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Developing  Ratings 


■  Define  an  arbitrary  mapping  from  2- 
dimensional  (L,C)  space  to  an  RI3 
“rating”  space 

■  From  previous  example 

■  Risk  1C 

■  Was  estimated  to  be 

■  C=4 

■  L=4  (highly  likely) 

■  Resultant  RI3  rating:  R1c=4 

■  Risk  1e 

■  Was  estimated  to  be 

■  C=5 

■  L=1  (not  likely) 

■  Resultant  RI3  rating:  R1e=3 

■  RI3  Rating  is  just  a  relative  ranking 

■  5  =  the  most  pressing 

■  1=  the  least  pressing,  but  not 
unimportant 

■  If  desired,  could  fall  back  to  a  scale  from 
1  to  3  (Green  to  Red) 


A  Desirable  Ratings  Map 
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Summary  Display  for  Unit  Under 

Evaluation 


■  Summary  display  for  decision  makers 

■  Uses  unique  2D->  1 D  mapping  of  (L,C)  to  ratings 

■  For  each  ‘ility,  display  the  worst  case  rating  of  any  risk 

■  Highlights  most  pressing  issues 

■  Complements  underlying  risk-methodology  data 

■  Invites  reader  to  investigate  further 

People,  Org.,  Skills 
Des.  Maturity  &  Stab. 

Scalability  &  Complexity 
Reliability 
Maintainability 
Software  Development 
Human  Factors 
Integrability 
Testability 

0  1  2  3  4  5 

Least  Pressing  p|g  RatjngS  Most  Pressing 
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Multiple  UUEs 


■  The  SPO  as  a  whole  can  look  across  UUEs 
■  Invites  drilling  down  for  more  detail 
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Multiple  UUEs  as  a  System  UUE 

Comparing  Components 


■  Look  across  components 

■  To  summarize  advancement  difficulty  for  lower  level  UUEs 

■  R(UUEa)  =  MaxThreads{l ntegrabi I itya,  Testability^  ...} 

■  R(UUEp)  =  MaxThreads{l  ntegrabi  I  ity|t,  Testability  |t, ...} 

■  System  program  manager  can  then  ask,  “which  of  my  components 
needs  the  most  help  today?” 

■  Colors  are  optional 
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Multiple  UUEs  as  a  System  UUE 

Comparing  Disciplines 


■  Look  across  threads 

■  To  summarize  advancement  difficulty  for  disciplines 

■  R(lntegrability)  =  MaxComps{lntegrabilitya,  lntegrabilityr,  ...} 

■  R(Testability)  =  MaxComps{Testabilitya,  Testabilityp,  ...} 

■  Perhaps  average  instead  of  max? 

■  System  program  manager  can  then  ask,  “how  are  my  processes 
working?” 

■  Consistent? 

■  Common  issues  faced  by  subsystems? 


Integrability 

Testability 

Reliability 

Maintainability 
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RI3  “Signature”  Evolves  over  Time 


■  RI3  can  be  used  to  support  risk  identification  both  in  support  of 
milestones  as  well  as  pre-MS  A  activity 

■  Input  to  PoPS 

■  Risks  could  actually  increase  as  more  knowledge  is  obtained 


Integrability 
Testability 
Reliability 
Maintainability 
Human  Factors 
Scalability  &  Complexity 
People,  Organization,  Skills 
Design  Maturity  &  Stability 


Integrability 
Testability 
Reliability 
Maintainability 
Human  Factors 
Scalability  &  Complexity 
People,  Organization,  Skills 
Design  Maturity  &  Stability 
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Uses  of  the  RI3  Methodology 


■  Part  of  the  usual  business  of  a  program  office,  XR,  AFRL,  or 
other  technology  developer 

■  As  an  input  to  their  risk/cost  methodology 

■  To  compare  and  evaluate  candidate  technologies  or 
concepts 

■  To  report  upwards  on  status  and  progress  (e.g.  PoPS) 

■  Other  potential  venues 

■  Pre-milestone  A  activities 

■  D&SWS:  LCM  Sufficiency  Reviews,  TD-1-13  Stage  Gating 

■  Labs,  contractors 

■  Independent  assessments:  e.g.  AFCAA,  guidance  for  red 
teams 

■  Source  selections,  Design  reviews,  etc... 
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RI3  Use  By  an  XR  or  SPO 

For  Risk  Management 


■  Incorporate  RI3  into  existing 
processes 

■  Risk  Management  Process 

■  SMC  guide:  “...process  can 
be  greatly  facilitated  by  the 
initial  identification  of 
categories  of  potential 
program  risk  initiating 
conditions...” 

■  Supports  Risk  Identification 

■  Helps  ensure  completeness  of 
technical  risks 

■  Resources  required 

■  No  additional  personnel  required 

■  May  add  as  little  as  2  hours  additional  work  to  determine  which  questions 
are  applicable 

■  Subsequent  work  is  part  of  normal  risk  processes 

■  Minimal  additional  training  required  beyond  risk  processes 

■  Workload  to  be  assessed  in  Fall  08  historical  test 


Risk 

Identification 


Risk 

Tracking 


Risk 

Mitigation 

Planning 


Risk  Mgmt  Guide  for  DoD 
Acquisition,  August  2006,  V  1.0. 

Similar  process  in  D&SWS  LCRM. 


<>  * 


Risk 

Mitigation 

Plan  Implementation 
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Usage  of  RI3  to  Feed  Risk 
Management  Processes 


RI3 

Guidebook 

Questions: 

•  Integration 

•  ilities 


Risks 


s 


5 

4 

3 

2 

1 

_________ 

1 

2 

3 

4 

5 

Tool 


Integrability 
Testability 
Reliability 
Maintainability 
Human  Factors 
Scalability  &  Complexity 
People,  Organization,  Skills 
Design  Maturity  &  Stability 


Active  Risk 
Manager 
(ARM) 
compatible 
file 


Summary  Displays 


Risk 

Management 

Step  2. 
Risk 

Identification 


Consequence 

Similar  output  for 

Tool 

cost  estimation 

Additional 

being  investigated 
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TD-1-12  Team  Members 


Name  Organization  Name  Organization 


Dorothy 

Arbiter 

Aerospace  Corp 

Joseph 

Baker 

AFMC/EN 

Greg 

Barnette 

AAC/EN 

Jim 

Bilbro 

NASA  (ret.) 

Manuel 

Casipit 

SMC/XR 

Tom 

Christian 

ASC/EN 

Toby 

Edison 

ESC 

Bob 

Frueholz 

Aerospace  Corp 

Suzanne 

Garcia 

CMU/SEI 

Peter 

Hantos 

Aerospace  Corp 

Jim 

Morgan 

AFRL  Mantech 

Bill 

Nolte 

AFRL/RY 

Paul 

Phister 

AFRL/RI 

Duane 

Sauve 

Ogden  ALC 

George 

Sarmiento 

AFMC/A5S 

Marc 

Smith 

AFMC/A2 

Mark 

Wilson 

SAF/AQR 

Kyle 

Yang 

MIT  Lincoln  Lab 

Larry 

Butterbaugh 

AFRL/RHOX 

Stacey 

Carswell 

AFRL/RWMI 

Bob 

Matthews 

AFRL/RYZC 

John 

Mistretta 

AFMC/EN 

Dieter 

Multhopp 

AFRL/RBAA 

John 

Remen 

AFRL/RZS 

Peter 

Roberts 

AFRL/RDHE 

Donna 

Senft 

AFRL/RVSV 

Mike 

Sorial 

AAC 

Charles 

Skira 

AFRL/RZTP 

Linda 

Taylor 

SMC/E  A 

Jim 

Jeter 

Warner  Robbins  ALC 

Art 

Chin 

SMC 
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Summary  &  Discussion 


■  TD-1  -12  consists  of 

■  Training  for  TRL  Assessments,  MRAs,  future  RI3 

■  Software  TRL  definition  clarification 

■  Risk  Identification:  Integration  &  ‘ilities  (RI3)  methodology 

■  Road  Ahead 

■  RI3  should  be  ready  for  external  application  in  January  09 

■  Engagments  on  D&SWS  Pilot  Projects  should  commence 

■  Potential  RI3  implications  and  touchpoints 

■  Risk  Management 

■  Cost  Analysis 

■  Systems  Engineering 

■  Potential  future  policy  implications  for  AF  not  yet  determined 

■  Questions  /  Comments? 
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Backups  Follow 
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Relationship  to  OSD  Checklists 


■  Various  OSD  checklists  are  available  on  the  DAU  website 

■  TRA  -  deals  primarily  with  setting  up  for  a  TRA,  not  how  to  conduct  a  TRA 

■  PRR  -  in  discussion  with  MRA  team 

■  PDR,  CDR 

■  Only  Navy  NAVAIR  appears  to  use  the  checklists 

■  Observations  on  checklists 

■  Checklists  are  excellent  sets  of  questions 

■  Checklists  are  much  broader  in  scope  than  RI3 

■  But 

■  Checklists  are  too  long  for  day-to-day  usage  by  a  SPO 

■  800+  questions:  if  everything  is  important,  then  nothing  is  important 

■  Checklists  use  a  non-standard  risk  definition  set  (1 -dimensional) 

■  Comments  on  RI3 

■  RI3  cherry-picked  &  derived  some  questions  from  checklists 

■  RI3  emphasizes  some  issues  that  checklists  appear  to  leave  out,  e.g.  Skills 
of  developers  (not  just  maintainers) 

■  RI3  more  geared  for  internal  SPO  usage  than  checklists  are 

■  RI3  geared  toward  feeding  internal  SPO  risks 

■  More  open-ended  questions,  geared  toward  promoting  best  detailed  practices,  as 
opposed  to  checking  the  box:  “Do  you  have  a  SEP?” 
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New  Criteria  to  Cover  Integration  /  ‘ilities 
Overview/Description 


Milestones  / 
KDPs 


A 


TRL-Driven,  Demonstration 
focused 


Enhanced  approach  will 
focus  S&T  provider  and 
Program  Office  attention  to 
issues  wrt  integration  and 
other  ‘ilities. 

Drives  risk  management. 


MRA  start  is  tied  more  to 
achieving  TRL  3  than  any 
other  criteria. 

Drives  risk  management. 


TRL  3/4 


TRL 


TRLs  are  necessary  but 
not  sufficient. 

TRA  provides 
“Snap-Shot  in  Time” 


Tech-Driven,  Integration  /  ‘ility  /  Risk  Focused 


MRA 
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Ratings  versus  Colors 


■  These  ratings  could  be  thought  of  as  creating  2  intermediate  colors 

■  Red/Yellow 

■  Reduces  tendency  to  try  to  avoid  high  numbers  because  they’re  red 

■  Green/Yellow 

■  Reduce  items  that  get  ignored  because  they’re  green 


Original  Colors 


1  2  3  4  5 


Proposed  RI3  Ratings 


2 

3 

4  4 

5 

2 

3 

3  4 

4 

2 

2 

3  3 

4 

1 

2 

2  3 

3 

1 

1 

2  2 

3 

1  2  3  4  5 
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A  Glimpse  at  a  Potential  RI3  Tool 

Instantiation 


llities  Drivers 

1 

2 

3 

4 

5 

Software 

Reliability 

Maintainability 

Testability 

Human  Factors 

Inteqrability 

Seal.  &  Complex. 

Peop.  Orqs  &  Sk. 

Des.  Maturity  &  Stab 

Clear  Chart 

Clear  Project  Information 


Program: 

Test  Case 

Date: 

8/1/2008 

UUE: 

Subsystem  A 

WBS  #: 

_ 1_2 _ 

Software  Development 

si 

Will  engineering  hardware  models  or  prototypes  be  available  for  software  testing  in  the  appropriate 
time  frame? 

S2 

Have  mechanisms  or  forums  been  established  to  ensure  appropriate  interactions  between 
simultaneously  working  software  development  teams? 

S3 

Have  mechanisms  or  forums  been  established  to  ensure  appropriate  interactions  between  the 
simultaneously  working  hardware  and  software  development  teams? 

S4 

Has  the  hardware/  software  interaction  been  simplified  to  the  maximum  extent? 

S5 

Has  the  interoperability  of  reuse/OTS  software  with  both  internal  and  external  system  elements  been 
considered? 

S6 

Has  the  ability  of  reuse/OTS  software  to  provide  required  safety,  security,  and  privacy  been 
confirmed? 

S7 

Has  the  ability  of  reuse/OTS  software  to  isolate  faults  in  the  integrated  reuse/OTS  been  confirmed? 
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A  Sample  Tool: 

Excerpt  from  DoD  PRR  Checklist 


Legend:  1  R 

G 

u 

N 

Item 

Comments  /  Mitigation 

1.  Engineering  and  Product  Design 

: 

o 

0 

0 

i 

Technical  Documentation,  Systems 
Integration,  and  Coordination 

a.  Technical  Documentation,  Systems  Integration, 
and  Coordination 

0 

0 

0 

0 

0 

l.a 

(1)  Are  the  contractor’s  engineering  drawings  and 
documents  complete  for  describing  the 
equipment  and  the  applicable  software  to  be 
delivered  under  this  program? 

1  a(1) 

K 

(2)  Are  there  provisions  to  assure  that  obsolete 
drawings  are  removed  and  discarded? 

1.a(2) 

(3)  Are  there  procedures  to  assure  that  all 
engineering  drawings  are  consistently 
prepared  and  that  all  revisions  and  Class  1 
engineering  changes  are  incorporated  into 
the  drawings? 

1a(3) 

•  RI3  could  be  similar,  but  prefer  to  leverage  existing  AFRL  effort  to  upgrade  TRL 
and  MRL  calculators  to  be  web-based  questionnaires 
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Likelihood  -  DoD  Guide 


LEVEL 

LIKELIHOOD 

PROBABILITY  OF 
OCCURRENCE 

1 

Not  Likely 

1  %-20% 

2 

Low  Likelihood 

21%-40% 

3 

Likely 

41%-60% 

4 

Highly  Likely 

61%-80% 

5 

Near  Certainty 

81  %-99  % 
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Consequence  -  Performance 


DoD  Guide 

Proposed  AF  Definition 

1 

Minimal  or  no  consequence  to  technical 
performance 

Minimal  consequence  to  technical  performance  but 
no  overall  impact  to  the  program  success.  A 
successful  outcome  is  not  dependent  on  this  issue; 
the  technical  performance  goals  will  still  be  met. 

2 

Minor  reduction  in  technical 
performance  or  supportability,  can  be 
tolerated  with  little  or  no  impact  on 
program 

Minor  reduction  in  technical  performance  or 
supportability,  can  be  tolerated  with  little  impact  on 
program  success.  Technical  performance  will  be 
below  the  goal  but  within  acceptable  limits. 

3 

Moderate  reduction  in  technical 
performance  or  supportability  with 
limited  impact  on  program  objectives 

Moderate  shortfall  in  technical  performance  or 
supportability  with  limited  impact  on  program 
success.  Technical  performance  will  be  below  the 
goal,  but  approaching  unacceptable  limits. 

4 

Significant  degradation  in  technical 
performance  or  major  shortfall  in 
supportability;  may  jeopardize  program 
success 

Significant  degradation  in  technical  performance  or 
major  shortfall  in  supportability  with  a  moderate 
impact  on  program  success.  Technical  performance 
is  unacceptably  below  the  goal. 

5 

Severe  degradation  in  technical 
performance;  Cannot  meet  KPP  or  key 
technical/supportability  threshold;  will 
jeopardize  program  success 

Severe  degradation  in  technical/supportability 
threshold  performance;  will  jeopardize  program 
success;  or  will  cause  one  of  the  triggers  listed 

below 
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Mandatory  Technical  Performance 
Consequence  Category  5  Triggers 


■  Any  root  cause  that,  when  evaluated  by  the  cross-functional 
team,  has  a  likelihood  of  generating  one  of  the  following 
consequences  must  be  rated  at  Consequence  Level  Five: 

■  Will  not  meet  KPP 

■  CTE  will  not  be  at  TRL  4  at  MS/KDP  A 

■  CTE  will  not  be  at  TRL  6  at  MS/KDP  B 

■  CTE  will  not  be  at  TRL  7  at  MS/KDP  C 

■  CTE  will  not  be  at  TRL  8  at  the  Full-rate  Production  Decision 
point 

■  MRL  will  not  be  at  8  by  Milestone  C 

■  MRL  will  not  be  at  9  by  Full-rate  Production  Decision  point 

■  System  availability  goal  will  not  be  met 
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Consequence  -  Schedule 


LEVEL 

DoD  Guide 

AF  Definition 

1 

Minimal  or  no  impact 

Negligible  schedule  slip 

2 

Able  to  meet  key  dates. 

Slip  <  *  month(s) 

Schedule  slip,  but  able  to  meet  key 
dates  (e.g.  PDR,  CDR,  FRP,  FOC) 

3 

Minor  schedule  slip.  Able  to  meet 
key  milestones  with  no  schedule 
float. 

Slip  <  *  month(s) 

Sub-system  slip  >  *  month(s)  plus 
available  float 

Schedule  slip  that  impacts  ability  to 
meet  key  dates  (e.g.  PDR,  CDR,  FRP, 
FOC) 

4 

Program  critical  path  affected. 

Slip  <  *  months 

Will  require  change  to  program  or 
project  critical  path. 

5 

Cannot  meet  key  program 
milestones. 

Slip  >  *  months 

Cannot  meet  key  program  or  project 
milestones. 

Integrity  -  Service  -  Excellence 


40 


40 


Consequence  -  Cost 


LEV 

EL 

DoD  Guide 

AF  Definition 

1 

Minimal  or  no 
impact 

For  A-B  Programs:  <5%  increase  from  last  approved  cost  estimate 

For  Post-B  Programs:  <X%  increase  in  PAUC  or  APUC  from  last  approved  cost  estimate 

or  program  cost  baseline 

2 

Budget  increase  or 
unit  production  cost 
increases. 

<  **  (1  %  of  Budget) 

For  A-B  Programs:  <10%  but  >5%  increase  from  last  approved  cost  estimate 

For  Post-B  Programs:  <1%  but  greater  than  X%  increase  in  PAUC  or  APUC  from  last 
approved  cost  estimate  or  program  cost  baseline 

3 

Budget  increase  or 
unit  production  cost 
increase 

<  **  (5%  of  Budget) 

For  A-B  Programs:  <15%  but  >10%  increase  from  last  approved  cost  estimate 

For  Post-B  Programs:  <5%  but  greater  than  1%  increase  in  PAUC  or  APUC  from  last 
approved  cost  estimate  or  program  cost  baseline 

4 

Budget  increase  or 
unit  production  cost 
increase 

<  **  (10%  of  Budget) 

For  A-B  Programs:  <20%  but  >15%  increase  from  last  approved  cost  estimate 

For  Post-B  Programs:  <10%  but  greater  than  5%  increase  in  PAUC  or  APUC  from  last 
approved  cost  estimate  or  program  cost  baseline 

For  O&S  Programs  and  Sustainment  Activities: 

5 

Exceeds  APB 
threshold 

>  **  (10%  of  Budget) 

For  A-B  Programs:  >=20%  increase  from  the  MS  A  approved  cost  estimate 

For  MS  Post  B  Programs:  >=10%  increase  in  PAUC  or  APUC  from  last  approved  cost 
estimate  or  program  cost  baseline  (in  danger  zone  for  Nunn-McCurdy  Breach) 
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MRL  Definitions 


MRL  1 

MRL  2 

MRL  3 

MRL  4 

MRL  5 

MRL  6 

MRL  7 

MRL  8 

MRL  9 

MRL  10 

Mfg 

Mfg 

Mfg 

Capability  to 

Capability  to 

Capability  to 

Capability  to 

Pilot  line 

Low  rate 

Full  rate 

feasibility 

concepts 

concepts 

produce  the 

produce 

produce  a 

produce 

capability 

production 

production 

assessed 

defined 

developed 

technology 

prototype 

prototype 

systems, 

demonstrated. 

demonstrated. 

demonstrated 

in  a 

components 

system  or 

subsystems  or 

Ready  to 

Capability  in 

and  lean 

laboratory 

in  a 

subsystem 

components  in 

begin  low  rate 

place  to  begin 

L4  I  1  \A  1  V/L4I  1 

nrnHi 

environment 

production 

in  a 

a  production 

production 

full  rate 

pi  UUUUUUI 1 

relevant 

production 

representative 

production 

practices  in 

environment 

relevant 

environment 

place 

- A 

L— 

environme^ 

L— 

- A 

— 

•  Production  relevant  environment  -  An  environment  normally  found  during  MRL  5  and  6  that  contains  key 
elements  of  production  realism  not  normally  found  in  the  laboratory  environment  (e.g.  uses  production 
personnel,  materials  or  equipment  or  tooling,  or  process  steps,  or  work  instructions,  stated  cycle  time,  etc.). 
May  occur  in  a  laboratory  or  model  shop  if  key  elements  or  production  realism  are  added. 

♦Production  representative  environment  -  An  environment  normally  found  during  MRL  7  (probably  on  the 
manufacturing  floor)  that  contains  most  of  the  key  elements  (tooling,  equipment,  temperature,  cleanliness, 
lighting,  personnel  skill  levels,  materials,  work  instructions,  etc)  that  will  be  present  in  the  shop  floor  production 
areas  where  low  rate  production  will  eventually  take  place. 

♦Pilot  line  environment  -  An  environment  normally  found  during  MRL  8  in  a  manufacturing  floor  production  area 
that  incorporates  all  of  the  key  elements  (equipment,  personnel  skill  levels,  materials,  components,  work 
instructions,  tooling,  etc.)  required  to  produce  production  configuration  items,  subsystems  or  systems  that  meet 
design  requirements  in  low  rate  production.  To  the  maximum  extent  practical,  the  pilot  line  should  utilize  rate 
production  processes. 
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